An Empirical Evaluation of Noise Contrastive Estimation for the Neural Network Joint Model of Translation

The neural network joint model of translation or NNJM (Devlin et al., 2014) combines source and target context to produce a powerful translation feature. However, its softmax layer necessitates a sum over the entire output vocabulary, which results in very slow maximum likelihood (MLE) training. This has led some groups to train using Noise Contrastive Estimation (NCE), which side-steps this sum. We carry out the first direct comparison of MLE and NCE training objectives for the NNJM, showing that NCE is significantly outperformed by MLE on large-scale ArabicEnglish and Chinese-English translation tasks. We also show that this drop can be avoided by using a recently proposed translation noise distribution.